Markov Random Fields for Super-Resolution
| William T. Freeman | Ce Liu |
| Massachusetts Institute of Technology | Microsoft Research New England |
This is an implementation of the example-based super-resolution algorithm of [1]. Although the applications of MSFs have now extended beyond example-based super resolution and texture synthesis, it is still of great value to revisit this problem, especially to share the source code and examplar images with the research community. We hope that this software package can help to understand Markov random fields for low-level vision, and to create benchmark for super-resolution algorithms.
When you refer to this code in your paper, please cite the following book chapter:
W. T Freeman and C. Liu. Markov Random Fields for Super-resolution and Texture Synthesis. In A. Blake, P. Kohli, and C. Rother, eds., Advances in Markov Random Fields for Vision and Image Processing, Chapter 10. MIT Press, 2011. To appear.
Algorithm
The core of the algorithm is based on [1]. We collect pairs of low-res and high-res image patches from a set of images as training. An input low-res image is decomposed to overlapping patches on a grid, and the inference problem is to find the high-res patches from the training database for each low-res patch. We use the kd-tree algorithm, which has been used for real-time texture synthesis [2], to retrieve a set of high-res, k-nearest neighbors for each low-res patch. Lastly, we run a max-product belief propagation (BP) algorithm to minimize an objective function that balances both local compatibility and spatial smoothenss.
Examples
Several examples of applying the example-based super resolution code in the package are shown below. These examplar images are also included in the package. Once you run the code, it should give you the same result.
We first apply bicubic sampling to enlarge the input image (a) by a factor of 4 (b), where image details are missing. If we use the nearest neighbor for each low-res patch independently, we obtain high-res but noisy results in (c). To address this issue, we incorporating spatial smoothness into a Markov Random Fields formulation by enforcing the synthesized neighboring patches to agree on the overlapped areas. Max-product belief propagation is used to obtain high-res images in (d). The inferred high-frequency images are shown in (e), and the original high-res are shown in (f).
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
| (a) Input | (b) Bicubic upsampling x 4 | (c) Independent NN | (d) Belief Propagation | (e) Inferred high frequency | (f) Original high-res |
| Some test images (selected from the Berklely Image Segmentation database [4]) that are used in our example-based super resolution package. | |||||
Examples from earlier publications
This is the low-resolution original image, processed output, and the true high-resolution image, for the "Maddie" image from reference [1]. The blurring and subsampling to create to low-resolution original image was done as described in [1].
 |
 |
 |
Algorithm
The core of the algorithm is based on [1, 2]. We collect pairs of low-res and high-res image patches from a set of images as training. An input low-res image is decomposed to overlapping patches on a grid, and the inference problem is to find the high-res patches from the training database for each low-res patch. We use the kd-tree algorithm, which has been used for real-time texture synthesis [3], to retrieve a set of high-res, k-nearest neighbors for each low-res patch. Lastly, we run a max-product belief propagation (BP) algorithm to minimize an objective function that balances both local compatibility and spatial smoothenss.
Usage
The package is written in MATLAB. We downloaded a kd-tree software package from http://www.mathworks.com/matlabcentral/fileexchange/2151 for our software, and this kd-tree package is included in our package as well. We have tested our system in Windows 7 x64, Mac OS X 10.6.5 and Debian GNU/Linux. After you download the package, launch MATLAB, change the current directory to #root/SRmatlab/Matlab/ (#root is where you unzipped the package), and run
GenerateTraining
You will see that the code attempts to compile the cpp files for the kd-tree package and then extracts patches from the training images. After the training is done, run
SuperResolution
to super resolve the test images using the training patches. It may take a while to build the kd-tree, so please be patient. If you set setting.IsKdtree=false in setEnvironment.m, then the kd-tree will not be called in searching for the neighbors for the patches in the test images. You may want to do this especially when MATLAB cannot successfully compile the enclosed kd-tree package.
The training images can be found at #root/SRmatlab/Database/Berkeley/Training, and the test images at #root/SRmatlab/Database/Berkeley/Test. You can easily create your own training and test databases under a different subfolder than Berkeley under a similar structure. Please change the following line
setting.Path.Database = fullfile(setting.Path.Home,'Database','Berkeley');
in setEnvironment.m to set the database path to the one you created. Rerun the two MATLAB functions GenerateTraining.m and SuperResolution.m to test how the system performs on different datasets.
Please contact Ce Liu should you have any questions or find any bugs.
Enjoy!
References
| [1] | W. T. Freeman, E. C. Pasztor, and O. T. Carmichael. Learning Low-Level Vision. International Journal of Computer Vision, 40(1), pp. 25-47, 2000. pdf |
| [2] | W. T. Freeman and Egon C. Pasztor. Learning Low-Level Vision. Appeared in IEEE Intl. Conf. on Computer Vision, Corfu, Greece, 1999. pdf |
| [3] | L. Liang, C. Liu, Y. Xu, B. Guo, and H. Shum. Real-time texture synthesis by patch-based sampling. ACM Transactions on Graphics, 2001. |
| [4] | D. Martin, C. Fowlkes, D. Tal and J. Malik. A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. In Proc. International Conference on Computer Vision, pp. 416-423, 2001 |
Last update: Jan 9, 2011